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Abstract 



We give an effective version of a result reported by Serre asserting 
that the images of the Galois representations attached to an abelian 
surface with End(vl) = Z large as possible for almost every 

prime. Our algorithm depends on the truth of Serre's conjecture for 
two dimensional odd irreducible Galois representations. Assuming 
this conjecture we determine the finite set of primes with exceptional 
image. We also give infinite sets of primes for which we can prove 
(unconditionally) that the images of the corresponding Galois repre- 
sentations are large. We apply the results to a few examples of abelian 
surfaces. 

Mathematics Subject Classification: 11F80, 11G10 
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1 Introduction 



Let A be an abelian surface defined over Q with End (A) := Endjj(A) = Z. 
Let pe : Gq — > GSp(4, Z e ) be the compatible family of Galois representations 
given by the Galois action on T e (A) = A[£°°](Q), the Tate modules of the 
abelian surface (we are assuming that A is principally polarized). Each pe is 
unramified outside £N, where N is the product of the primes of bad reduction 
of A. If we call Ge°° the image of pe then we have the following result of Serre 
(cf. [Se 86]): 

Theorem 1.1 If A is an abelian surface over Q with End(A) = Z and 
principally polarized, then Ge°° = GSp(4, 7he) for almost every £. 

Remark: If Ge is the image of pe, the Galois representation on ^-division 
points of A(Q) (and the residual mod £ representation corresponding to pe), 
it is enough to show that Ge = GSp(4, F^) for almost every £ (cf. [ Se 86]). 
Serre proposed the problem of giving an effective version of this result: 
"...partir de courbes de genre 2 explicites, et tcher de dire partir de quand 
le groupe de Galois correspondant Ge devient gal GSp(4, F^) ". But Serre's 
proof depends on certain ineffective results of Faltings and therefore does not 
solve this problem. 

In this article, we will give an algorithm that computes a finite set JF of 
primes containing all those primes (if any) with image of the corresponding 
Galois representation exceptional, i.e., different from GSp(4, F^). The valid- 
ity of our method depends on the truth of Serre's conjecture for 2-dimensional 
irreducible odd Galois representations, conjecture (3.2.4?) in [Se 87]. This 
means that if £ is a prime such that Ge ^ GSp(4, F^) and £ J 7 , then Ge has 
a 2-dimensional irreducible (odd) component that violates Serre's conjecture. 

The method is inspired by the articles [Se 72], [Ri 75], [Ri 85] and [Ri 97] 
where the case of 2-dimensional Galois representations is treated. 

In the examples, we also give infinite sets of primes for which we can prove 
the result on the images unconditionally, i.e., without assuming Serre's con- 
jecture. Results of this kind were previously obtained by Le Duff only under 
the extra assumption of semiabelian reduction of the abelian surface at some 
prime. Our technique has two advantages: it does not have any restriction 
on the reduction type of the abelian surface, and in the case of semiabelian 
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reduction it allows us to prove the result on the images (unconditionally) for 
larger sets of primes. 

1 want to thank N. Vila, A. Brumer and J-P. Serre for useful remarks and 
comments. 

2 Main Tools 

2.1 Maximal subgroups of PGSp(4,F^) 

In [Mi 14], Mitchell gives the following classification of maximal proper sub- 
groups G of PSp(4, F^) (£ odd), as groups of transformations of the projective 
space having an invariant linear complex: 

1) a group having an invariant point and plane 

2) a group having an invariant parabolic congruence 

3) a group having an invariant hyperbolic congruence 

4) a group having an invariant elliptic congruence 

5) a group having an invariant quadric 

6) a group having an invariant twisted cubic 

7) a group G containing a normal elementary abelian subgroup E of order 
16, with: G/E = A 5 or S 5 

8) a group G isomorphic to A e , S e or A 7 . 

For the relevant definitions see [Hi 85], see also [Bl 17] and [Os 77] for cases 
7) and 8). 

Remark: This classification is part of a general "philosophy" : the subgroups 
of GL(n, F^), £ large, are essentially subgroups of Lie type, with some excep- 
tions independent of £ (see [Se 86]). 

From this we obtain a classification of maximal proper subgroups H of 
PGSp(4, F^) with exhaustive determinant. It is similar to the above one, 
except that cases 7) and 8) change according to the relation between H and 
G, given by the exact sequence: 

1 -> G -> H -> {±1} -> 1. 
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2.2 The action of Inertia 



From now on we will assume that I is a prime of good reduction for the abelian 
surface A. Then it follows from results of Raynaud that the restriction pe\i i 
has the following property (cf. [Ra 74], [Se 72]): 



Theorem 2.1 If V is a Jordan-Hlder quotient of the Ig-module A[£](Q) of 
dimension n over Fg, then V admits an Fin -vector space structure of dimen- 
sion 1 such that the action of lg on V is given by a character <fi : — > F|„ 
(t stands for tame) with: 

= 0?\...<^, (2.1) 

where fa are the fundamental characters of level n and di = or 1 , for every 
% — 1, 2, n. 

All this statement is proved by Serre in [Se 72] except for the bound for the 
exponents, which is the result of Raynaud mentioned above, later generalized 
by Fontaine-Messing. 

We will use the following lemmas repeatedly (cf. [Di 1901]): 

Lemma 2.2 Let M G Sp(4, F) be a symplectic transformation over a field F . 
The roots of the characteristic polynomial of M can be written as a, (3, a -1 , 
for some a, f3. 

Remark: A similar result holds in general for the groups Sp(2n, F). 

In the case of the Galois representations attached to A, we know that det(p^) = 

X 2 , where x is the mod i cyclotomic character. Therefore, we obtain: 

Lemma 2.3 The roots of the characteristic polynomial of p^(Frob p) G Ge 
can be written as a, (3,p/a,p/f3 (p \ IN). 

Remark: Here Frob p denotes the (arithmetic) Frobenius element, defined up 
to conjugation. The value of the representation in it is well defined precisely 
because of the fact that the representation is unramified at p. 



Proof: Use lemma pT2| , G e C GSp(4,F£), and the exact sequence: 



1 -> Sp(4,F,) -> GSp(4,F € ) -> F* -> 1. 
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Remark: The same is true for p^Frob p) G Ggoo. Thus, the characteristic 
polynomial of /^(Frob p) has the form: 

x 4 — a p x 3 + b p x 2 — pa p x + p 2 

with a p , bp G Z, a p = trace(p^(Frob p)). 

From formula (2.1), we obtain the following possibilities for pe\ I( (£\ N): 

( 1 * * * \ / tp 2 * * \ 

Ox** ip2 * * 

1*' ^ 2 o ' 

\o xj \o o o ipi) 

(fa o * *\ /^ +£2 o \ 
^2 * * o < +£3 

1*' o ijf +1 

\ xj \ o o vl + V 

where ^ is a fundamental character of level i. 



3 Study of the Images 

3.1 Reducible case: 1-dimensional constituent 

Let c be the conductor of the compatible family {pe}. For the case of the 
jacobian of a genus 2 curve, it can be computed using an algorithm of Q. 
Liu except for the exponent of 2 in c, which can easily be bounded using the 
discriminant of an integral model of the curve (see [Liu 94]). 

Suppose that the representation pg is reducible with a 1-dimensional sub (or 
quotient) representation given by a character \x. This character is unrami- 
fied outside £N and takes values in F^, therefore from the description of pt\i t 
given in section 2.2 we have p, = ex\ with e unramified outside iV and i — 
or 1. Clearly cond(e) | c. After semi-simplification, we have: 

Pi e X l © 7T, 
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for a 3- dimensional representation tt with det(7r) = e~ x x 2 ~ % ■ 
Therefore, cond(e) 2 | c. Let d be the maximal integer such that d 2 | c. If 
we take a prime p = 1 (mod d), we have e(p) = 1 so is a ro °t of the 
characteristic polynomial of /^(Frob p). This gives: 

b p - a p (p+l)+p 2 + 1 = (mod£), (3.1) 



both for % = and z = 1 (in agreement with lemma |2.3|) . 
By the Riemann hypothesis, the roots of the characteristic polynomial of 
P£(Frob p) have absolute value v /p. This gives automatically bounds for the 
absolute values of the coefficients a p and b p , and from these bounds we see 
that for large enough p congruence (3.1) is not an equality. Therefore, only 
finitely many primes t may verify (3.1). 

Variant: Instead of taking a prime p = 1 (mod d), we can work in gen- 
eral with p of order / in (Z/gK)*. Let Pol p {x) be the characteristic polyno- 
mial of p£(Frob p). Then e{p)p l is a root of Pol p (x), with i — or 1, and 
e(p)/ = 1 g F f . Then 

Res(Pol p (x), x s - 1) = (mod £) (3.2) 

where Res stands for resultant (again, cases i = and 1 agree). 

This variant is used in the examples to avoid computing Pol p (x) for large p. 



3.2 Reducible case: "related" 2-dimensional 
constituents 

Suppose that, after semi-simplification, pi decomposes as the sum of two 2- 
dimensional irreducible Galois representations: pe = 7i"i © 7r 2 . Assume also 
that these two constituents are related by lemma ^73| , i.e., if a, [3 are the roots 
of the characteristic polynomial of 7Ti(Frob p), then p/a,p//3 are the roots of 
that of 7r 2 (Frob p). If not, then it follows from lemma |2.3j that a = p//3, so 
det(7Ti) = det(7r 2 ) = Xi this case will be studied in the next subsection. 
Using the description of pi\i t given in section 2.2 we see that one of the 
following must happen (where e is a character unramified outside N): 

• Case 1: det(7i"i) = ex 2 , det(7r 2 ) = e^ 1 . 

• Case 2: det(7i"i) = ex, det(7r 2 ) = e^x- 
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• Case 1: In this case we have the factorization: 

Pol p (x) = (x 2 — rx + p 2 e(p)) (x 2 + e^ 1 (p)) (mod £). 

ps yp) 

As in the previous subsection: cond(e) | d. Eliminating r from the equation, 
we obtain: 

Q p (b p ,a p ,e(p)) := (e(p)b p -l-p 2 e(p) 2 )(pe(p) + l) 2 -a 2 p pe(p) 2 = (mod £). 

If we take p = 1 (mod d), we obtain: 

(b p - 1 - p 2 ) (p + I) 2 = a 2 p p (modi). (3.3) 

Again, from the bounds for the coefficients we see that for large enough p 
this is not an equality. Thus, only finitely many £ can satisfy (3.3). 

Alternatively, for computational purposes, we may take p with p* = 1 
(mod d). Then we have: 

Res(Q p (b p , a p , x), x f - 1) = (mod £). (3.4) 

• Case 2: This case is quite similar to the previous one. We start with: 

PolJx) = (x 2 — rx + ps[p)) (x 2 -— + ps^ip)) (mod £) 

£(p) 

with cond(e) | d. From this: 

Q' p (b p , a p , e{p)) := (e(p)b p -p- pe(p) 2 )(e(p) + l) 2 - a 2 p e(p) 2 = (mod £). 
Thus, if p = 1 (mod d), 

A(b p -2p)=a 2 p (modi). (3.5) 
In general, if p* = 1 (mod <i), 

Res(Qp(6 p , a p , a;), a/ — 1) = (mod £). (3.6) 
In this case, the fact that this holds only for finitely many primes £ is non- 



trivial. It may be thought of as a consequence of theorem [LI 
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3.3 The remaining reducible case 



As explained above, in the remaining reducible case we have: p s e s = 7ri © 7r 2 
with det(7i"i) = det(7T 2 ) = x- We have described in section 2.2 the possibilities 
for pe\i e . This gives for Ttx\j t and 7T 2 |j € : 

1 * \ ffo 

x) or Vo ¥2 

Besides, cond(7ri)cond(7r 2 ) | c. 

At this point we invoke Serre's conjecture (3.2.4?) (see [Se 87]) that gives us 
a control on tti and 7r 2 . Both representations should be modular of weight 2, 
i.e., there exist two cusp forms fi, / 2 with: 

Ph,t = tti, P /2/ = vr 2 , /1 G S 2 (ATi), / 2 G S 2 (iV 2 ), 

N1N2 I c (we are assuming 7Ti,7r 2 to be irreducible; otherwise, they are cov- 
ered by section 3.1). Both cusp forms have trivial nebentypus. 



There are finitely many cusp forms in these finitely many spaces. So we 
have an algorithm to detect the primes i falling in this case by comparing 
characteristic polynomials mod £, due to 

We take all pairs of integers iV\, iV 2 with iViiV 2 = c and all pairs of cusp forms 
fi G S^iVi), / 2 G S' 2 (iV 2 ) (either newforms or oldforms). If we denote by 
Polf itP (x) the characteristic polynomial of p/^(Frob p) {i = 1,2), we should 
have for some such pair fi, f 2 : 

Polf 1:P (x)Polf 2tP (x) = Pol p (x) (mod £) (3.7) 

for every p \ £N. 

Theorem O] guarantees that this can only happen for finitely many primes. 



Remark: The Galois representations pf it £ attached to fi were constructed 
by Deligne (cf. [De 71]). The polynomials Polf uP (x) are of the form 

Polf iyP (x) = x 2 - c p x + p, 

where c p is the eigenvalue of fi corresponding to the Hecke operator T p . 
These eigenvalues, and a fortiori the characteristic polynomials Polf tP (x) for 
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any cusp form /, can be computed with an algorithm of W. Stein (cf. [St]). 
The compatible family of Galois representations constructed by Deligne, in 
the case of a cusp form / e S^iV), shows up in the jacobian Jq(N) of the 
modular curve X (N): it agrees with a two-dimensional constituent of the 
one attached to the abelian variety Af corresponding to /. 

For computational purposes, we introduce the following variant: observe that 
either JVi or N 2 (say Ni) satisfy 

Ni | c, JVi < Vc. 

Consider all divisors of c verifying this, maximal (among divisors of c) with 
this property. Call S the set of such divisors. Then we are supposing that 
there exists / G S 2 (t) with t £ S and 

Res(Pol fjP (x), Pol p {x)) = (mod £), 

for every p \ IN '. Therefore, for some t E S 

Res( Yl Polf,p{x),Pol p (x)) = (mod£), (3.8) 

fes 2 (t) 

for every p \ IN . 

With this formula we compute in any given example all primes t falling 
in this case. 

Remark: In all reducible cases (sections 3.1, 3.2 and 3.3) we have consid- 
ered reducibility over fy 

3.4 Stabilizer of a hyperbolic or elliptic congruence 

If Gi corresponds to an irreducible subgroup inside (its projective image) 
some of the maximal subgroups in cases 3) and 4) of Mitchell's classification, 
there is a normal subgroup of index 2 of Ge such that 

1 M e -> G e -> {±1} -> 1, 

and the subgroup Mi is reducible (not necessarily over F^). 

In fact, a hyperbolic (elliptic) congruence is composed of all lines meeting 
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two given skew lines in the projective three dimensional space over F^ defined 
over F^ (Wp, respectively), called the axes of the congruence (see [Hi 85]). 
The stabilizer of such congruences consists of those transformations that fix 
or interchange the two axes, and it contains the normal reducible index two 
subgroup of those transformations that fix both axes. 

From the description of pi\i t given in section 2.2 we see that if I > 3, it 
is contained in M^. Therefore, if we take the quotient Gg/Mi we obtain a 
representation Gq — > Ci whose kernel is a quadratic field unramified outside 



N. Then, there is a quadratic character . 
= — 1 =>■ PeiFiob p) is of the form: 

/0 * 



(Z/cZ) 



Co with: 





* 
* 



* 





* 


0/ 



Therefore, trace (pi (Frob p)) = , i.e., 







(3.9) 



(mod i\ 
for every p \ C.N with <fi(p) = — 1. 

Considering all quadratic characters ramifying only at the primes in N we 
detect the primes I falling in this case. 

Once again, from theorem 1.1 , it follows that this set is finite (of course, this 
fact strongly depends on the assumption End(A) = Z). 



3.5 Stabilizer of a quadric 

This case can be treated exactly as the one above: assuming again absolute 
irreducibility of the image Ge, it contains a normal subgroup of index 2, and 
we obtain a quadratic character unramified outside N verifying formula (3.9). 
In fact, in this case pi is the tensor product of two irreducible 2-dimensional 
Galois representations (see [Hi 85], page 28), one of them dihedral (this is the 
necessary and sufficient condition for the tensor product to be symplectic, 
see [B-R 89], page 51), so the matrices in Ge are of the form: 



/ av 





cv 


o\ 




f ° 


az 





cz \ 





a z 





cz 


or 


av 





cv 





bv 





dv 








bz 





dz 


Vo 


bz 





dz J 




y bv 





dv 





depending on the value of the quadratic character <p. 
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3.6 Stabilizer of a twisted cubic 



This case is incompatible with the description of pt\i t given in section 2.2. In 
fact, in this case all upper-triangular matrices are of the form (see [Hi 85], 
page 233): 



/a 3 


* 


* 







a 2 d 


* 


* 








ad 2 


* 


1° 








d 3 ) 



In no case is the subgroup of Gg given by pe\i e of this form. 

3.7 Exceptional cases 

The cases already studied cover all possibilities in the classification except 
the exceptional groups, i.e., cases 7) and 8). 

In these cases, comparing the exceptional group H C PGSp(4,F£) (its or- 
der and structure) with the fact that P(G^) contains the image of F(pe\i g ) 
described in section 2.2, we end up with the only possibilities (£ > 3): 

£ = 5,7. 

For these two primes, as for any prime we suspect of satisfying Gi ^ GSp(4, F^), 
we compute several characteristic polynomials Pol p (x) mod £. At the end, 
either we prove that it must be Ge = GSp(4,F^) (because the orders of 
the roots of the computed polynomials do not give any other option) or we 
reinforce our suspicion that £ is exceptional. 

3.8 Conclusion 

Having gone through all cases in the classification (the stabilizer of a parabolic 
congruence is reducible, it has an invariant line of the complex, cf. [Mi 14]) 
we conclude that for all primes I except those whose image, according to our 
algorithm, may fall in a proper subgroup (according to theorem |1 . 1| , only 
finitely many) the image of f(pi) is PGSp(4,F£). 

From this it easily follows that if £ is not one of the finitely many exceptional 
primes we have Gg = GSp(4,F^) and applying a lemma of [Se 86] (see also 
[Se 68]) we obtain Ge°° = GSp(4, Zg). 

Recall that at one step we have assumed the veracity of Serre's conjecture 
(3.2.4 ? ). 
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4 An example 



We have applied the algorithm to the example given by the jacobian of the 
genus 2 curve given by the equation: 

y 2 = x 6 - x 3 - x+ 1. 

The algorithm of Q. Liu computes the prime-to-2 part of the conductor. 
From this computation and the bound of the conductor in terms of the dis- 
criminant of an integral equation (cf. [Liu 94]) we obtain: c | 2 12 -23-5. 
We exclude a priori the primes dividing the conductor: 2, 5 and 23. 
We sketch some of the computations performed: 

• Reducible cases with 1-dimensional constituent or two related 2-dimensional 
constituents: 

The maximal possible value of the conductor of e is d = 64. We compute the 
characteristic polynomials of /^(Frob p) for the primes p = 229, 257, 641, 769 
and applying the algorithm (equations (3.2), (3.4) and (3.6)) we easily check 
that no prime £ > 3 falls in these cases. 

Remark: The characteristic polynomials used at this and the remaining steps 
can be found in section 6. 

• Remaining reducible case: 

First we describe the set of special divisors of c: 

S= {368,460,512,640}. 

Then we compute, for each t G S and each Hecke eigenform / e S2 (t), the 
characteristic polynomial Polf jP (x) for p = 3, 7, 11, 13, 17, 19 with the algo- 
rithm implemented by W. Stein ([St]). Then, comparing these polynomials 
with the characteristic polynomials of p^(Frob p) as in formula (3.8) we see 
that no prime I > 3 falls in this case. 

• Cases "governed" by a quadratic character: 

We have to consider all possible quadratic characters (f> unramified outside c 
(there are 15) and for each of them take a couple of primes p with <fi(p) = — 1 
and a p 7^ 0. Applying the algorithm (formula (3.9)) we see that no prime 
£ > 3 falls in these cases. At this step we have used the values a p for the 
primes p = 3, 7, 13, 97, 113, 569, 769. 

• Exceptional cases: 

We compute the reduction of a few characteristic polynomials modulo 7 and 
we find elements whose order (in PGSp(4, F 7 )) does not correspond to the 
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structure of any of the exceptional groups. 
From all the above computations we conclude: 

Theorem 4.1 Let A be the jacobian of the genus 2 curve: 

y 2 = x 6 -x 3 -x + 1. 

Let G^oo be the image of pa,i, the Galois representation on A[£°°](Q) 7 whose 
conductor divides 2 12 • 5 • 23. Then, assuming Serre's conjecture (3.2. 4?) it 
holds: 

= GSp(4,Z<) 

for every t > 5, £ ^ 23. 

Remark: We are not claiming that the image is not maximal for any of the 
four excluded primes. 

5 Unconditional Results and more examples 
5.1 The case of semiabelian reduction 

For certain genus 2 curves one can prove that the image is large for an infinite 
set of primes by using the following results of Le Duff [LeD 98]: 

Proposition 5.1 Let A be an abelian surface defined overQ. Suppose that 
for a prime p of bad reduction of A, A® (the connected component o/O in the 
special fiber of the Nron Model of A at p) is an extension of an elliptic curve 
by a torus. Then, for every prime i ^ p with i \ (number of connected 
components of A p ), Ge contains a transvection. 

Recall that a transvection is an element u such that Image {u — 1) has dimen- 
sion 1. 

Proposition 5.2 ([LeD 98]) If G C Sp(4, F^) is a proper maximal subgroup 
containing a transvection, all its elements have reducible (over ¥ e ) charac- 
teristic polynomial. Therefore, a transvection together with a matrix with 
irreducible characteristic polynomial generate Sp(4, F^). 
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Remark: We can also find in [Mi 14] the list of maximal subgroups of 
PSp(4, We) containing central elations, and a central elation is the image in 
PSp(4, We) of a transvection in Sp(4,F^). These groups correspond to cases 
1) and 3) in section 2.1 or to a group having an invariant line of the complex, 
defined over F^. 

Recall that Pol q {x) denotes the characteristic polynomial of /^(Frob q) for 
any prime q of good reduction for the abelian surface A and I ^ p. From the 
two previous results it follows: 

Theorem 5.3 (Le Duff) Let p be a bad reduction prime verifying the condi- 
tion of proposition [571] and q a prime with Pol q (x) irreducible, then for every 
£\2pq$(p) such that Pol q (x) is irreducible modulo I, Ge = GSp(4, F^). 
If A q is the discriminant of Pol q {x) and Aq 9 the discriminant of Q q (x) : = 
x 2 — a q x + b q — 2q the irreducibility condition reads: 

(A*) = _j and = _ L 



EXAMPLE (Le Duff): Take the genus 2 curve: 

C 2 \ y 2 = x 5 - x + 1. 

A 2 = J{C 2 ) has good reduction outside 2, 19, 151. For p = 19, 151 the con- 
dition in proposition [5]l] is satisfied with = 1. 

Take q = 3, Pol$(x) is irreducible and theorem |5T5| gives: Ge = GSp(4, F^) 
for every i > 3 with (^) = -1 and (|) = -1. 

Remark: Of course, considering more irreducible characteristic polynomi- 
als one can obtain the same result for other primes. In particular, Ge = 
GSp(4,F £ ) for £ = 19, 151 (cf. [LeD 98]). 

Remark: The example in the previous section also verifies Le Duff's condi- 
tion. 



Let us apply our method to this example. The invariants are: 

c = cond(v4 2 ) | 2 8 ■ 19 ■ 151 (computed with Liu's algorithm), then cond(£:) | 

d = 16; and the set S = {256, 604, 608}. 

In this example, we only have to worry about those maximal subgroups in 
Mitchell's classification containing central elations. Therefore, we only have 
to discard the maximal subgroups considered in sections 3.1, 3.3 and 3.4. 
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• The reducible case with 1-dimensional constituent is easily handled us- 
ing the characteristic polynomials (see section 6) Pol p (x) for p = 17, 97 and 
we conclude that no prime I > 2 falls in this case. 

• Due to the fact that the spaces of modular forms ^(t) for t G S are rather 
large, we decided to save computations and to apply the procedure described 
in section 3.3, formula (3.8), only to the prime p = 3. After computing 
all resultants of Pol$(x) with all the characteristic polynomials Polf$(x) for 
/ G 5*2 (t), t G S, we find the possibly exceptional primes £ > 2: 

£ = 3,5, 11, 19, 29, 31, 41, 61, 109, 151. 

Having computed the characteristic polynomials 

Pol p (x) for p = 11,41,79,101,199,211 

(see section 6) we checked that for each of the ten possibly exceptional primes 
I listed above one of these six polynomials is irreducible modulo I. Then, 
applying theorem |5.3| , we conclude that none of these primes is exceptional. 
Thus, no t > 2 has reducible image. 

• Cases governed by a quadratic character: we have to consider all possible 
quadratic characters <fi unramified outside c and for each of them take a 
couple of primes p with (f>(p) = —1 and a p ^ 0. We use the values a p for 
p = 3,5, 97, 257 (see section 6) and an application of the algorithm (formula 
(3.9)) proves that the only possibly exceptional primes i > 2 are: 

£ = 3,5,11,97,257. 

We already mentioned that 3, 5 and 11 are not exceptional. Applying theorem 
|BT3l again we see that 97 and 257 are also non-exceptional because Polu (x) 
is irreducible modulo 97 and Pohsi^x) is irreducible modulo 257. Then, we 
conclude: 

Theorem 5.4 Let A 2 be the jacobian of the genus 2 curve given by the equa- 
tion y 2 = x 5 — x + 1. Assume Serre's conjecture (3.2. 4? ) (cf. [Se 87]). Then 
the images of the Galois representations on the l-division points of A 2 are 

Gi = GSp(4,F^), for every t > 2. 

Remark: p 2 is also irreducible over F 2 . This irreducibility for all i is equiva- 
lent to the fact that A 2 is isolated in its isogeny class in the sense that any 
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abelian variety isogenous to A 2 over Q is isomorphic to A 2 over Q. Unfortu- 
nately, this condition of being isolated is not effectively verifiable. 

Among the subgroups containing central elations, we have used Serre's con- 
jecture only to eliminate the following one: 

G e C {A x B E GL(2,F £ ) x GL(2,F £ ) : det(A) = det(B) = x }- (*) 

Take q with Pol q (x) irreducible. If {-j 3 -) = —1 case (*) cannot hold, because 
the matrices A and B would have their traces in ¥p \ F^. This follows from 
the factorization: 

Pol,( X ) = («> - ( "< + f^ )x + q) (f - + «) ■ 

Then, again using Pol^{x) we prove without using Serre's conjecture the 
following 

Theorem 5.5 The images of the Galois representations on the l-division 
points of A 2 are 

G e = GSp(4,F £ ), for every £>3 with (-) = -1. 

Observe that we have obtained an unconditional result that is stronger than 
the one in [LeD 98], because it only uses the condition on one of the discrim- 
inants (thus, it applies to more primes). 

We warn the reader that there is a mistake in [LeD 98], pag. 521, the poly- 
nomial Poln corresponding to this example is wrongly computed. It should 
read: 

x 4 + 7x 3 + 3lx 2 + 77x + 121. 



5.2 Unconditional results in the general case 



We will show now that even in the case that the condition of proposition [O 
is not verified at any prime, we can obtain similar unconditional results. 
In an arbitrary example, if we do not use Serre's conjecture, there is another 
case to consider (in addition to case (*) ): 

Ge C {M G GL(2, F |2 ) : det(M) = X }- (**) 
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The inclusion of this group in GSp(4, F^) is given by the map: 

M -> diag(M, M Floh ), where Frob is the non-trivial element in Gal(F£2/F^). 

Two tricks allow us to discard this case: 

i) Suppose that for a prime q, Pol q (x) decomposes over Q as follows: 

Pol q (x) = (x 2 + Ax + q)(x 2 + Bx + q), A^B. 
Then case (**) cannot hold if I \ B — A and I ^ q. 

ii) Suppose that p 2k+1 || cond(v4), then for every £ \ p<&(p), case (**) 
cannot hold. The condition on i is imposed to ensure that for these I 
p2k+i || conc i(p^ a i so holds. 

EXAMPLE (Smart): The following curve is taken from the list given in 
[Sm 97] of all genus 2 curves defined over Q with good reduction away from 
2: 

C 3 : y 2 = x(x 4 + 32x 3 + 336x 2 + 1152a; - 64), 

A3 = J(Cz), c I 2 20 (this is the uniform bound for the 2-part of the conductor 
of abelian surfaces over Q, cf. [B-K 94]). Le Duff's method cannot be applied 



to this example; the condition of proposition |5.1| is not verified at 2. 
We eliminate ALL maximal proper subgroups in Mitchell's classification us- 
ing the characteristic polynomials Pol p (x) for several primes p and cond(£) | 
1024, S = {1024}, with the algorithm described in section 3. 
To be more precise, the reducible cases treated in section 3.1 and 3.2 are ex- 
cluded using the polynomials Pol p (x) for p = 3, 17, 19, 31. Assuming Serre's 
conjecture, the remaining reducible case is excluded using the polynomials 
Pol p (x) for p = 7, 11, 13. The cases considered in sections 3.4 and 3.5 are 
excluded using the polynomials Pol p (x) for p = 3,5. Finally, with the tech- 
nique described in section 3.7 we check that I = 5, 7 are non-exceptional. 
All characteristic polynomials used are listed in section 6. 
After these computations we find no exceptional primes. Then, we conclude 



Theorem 5.6 Assume Serre's conjecture (3.2.4-?) (cf. [Se 87]). Then the 
images of the Galois representations on the l-division points of A 3 are 



G e = GSp(4, W e ) , for every £ > 3. 
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Without Serre's conjecture, trick (i) is used to discard case (**). In fact, 
Pol§{x) decomposes as in (i) with A = — 2 and B = 0. The same happens to 
Pol 17 (x). 

To deal with case (*) we check that Pol 3 (x) is irreducible and Aq 3 = 12 (see 
section 6). We obtain the unconditional result: 

Theorem 5.7 The images of the Galois representations on the t-division 
points of A 3 are 



5.3 Further examples 

In [Le 91], Leprvost gives a genus 2 curve over Q(t) with 13-rational torsion. 
For t — 13 we obtain: 

C : y 2 = -4a; 5 + 300a; 4 - 1404a; 3 + 5408a; 2 - 8788a; + 28561, 

A = J(C) has cond(A) = 2 a • 13 3 • 5 2 • 17 2 . Le Duff's condition is not verified 
at any prime. We can determine the image as in the previous example, 
with or without assuming Serre's conjecture (in the "reducible case with 1- 
dimensional constituent" we find I = 13 as exceptional prime). 
Remark: Here trick (ii) eliminates case (**) for every £ ^ 13 because 



Brumer and Kramer (unpublished) have given examples of jacobians of genus 
2 curves with prime conductor. For them our algorithm determines the image 
with just a few computations. For instance, when applying Serre's conjecture 
no computation is necessary because we have S — {1} and ^(l) = 0. 
One of these examples is given by the jacobian of the genus two curve: 

C : y 2 = x(x 2 + 1) (1729a; 3 + 45568a; 2 + 25088a; - 76832). 

The conductor of J(C) is 709. 

Remark: All the examples of abelian surfaces considered in this article ver- 
ify the condition End(A) = Z. This follows in particular from our result 
on the images of the attached Galois representations (the condition on the 
endomorphism algebra is also necessary for this result to hold). 



Ge = GSp(4, F^), for every £>3 with 




-1. 



13 3 || cond(A) and $(13) = 13. 
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6 Computed characteristic polynomials 



We list all the characteristic polynomials Pol p (x) that have been used in the 
examples of the abelian surface A in section 4 and the abelian surfaces A 2 
and A 3 in section 5. 

Recall that in any case the polynomial Pol p (x) is of the form 

x A — a p x 3 + b p x 2 — pa p x + p 2 
so it is enough to give the values a p , b p . 

Abelian surface A (section 4): 



p 


dp 


b p 


3 


-3 


6 


7 


-2 


6 


11 


-4 


18 


13 


-5 


16 


17 





22 


19 


-6 


42 


97 


6 


154 


113 


18 


250 


229 


24 


534 


257 


15 


148 


569 


6 


-118 


641 


12 


-266 


769 


-6 


402 
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Abelian surface A 2 = 



J(C 2 ) (section 5.1): 



V 


a p 


bp 


3 


-3 


7 


5 


-5 


15 


17 


-3 


16 


97 


-8 


86 


257 


-11 


-113 


11 


-7 


31 


41 


-7 


72 


79 


7 


75 


101 


-8 


-16 


199 


25 


338 


211 


-17 


103 


281 


1 


148 



Abelian surface A 3 = J(C 3 ) (section 5.2): 



V 


dp 


6 P 


3 


2 


4 


5 


2 


10 


7 


-2 


2 


11 


-2 


12 


13 


-6 


18 


17 


4 


22 


19 


-2 


-4 


31 


4 


46 
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